Field of the Invention
The invention relates to a Mixed Media Reality (MMR) system that uses printed media in combination with electronic media to retrieve mixed media documents. Still more particular, the present invention relates to a method for delivering targeted information to mobile devices using an MMR system and a user's context, history and pattern of interacting with paper documents.
Background of the Invention
Document printing and copying technology has been used for many years in many contexts. By way of example, printers and copiers are used in commercial office environments, in home environments with personal computers, and in document printing and publishing service environments. However, printing and copying technology has not been thought of previously as a means to bridge the gap between static printed media (i.e., paper documents), and the “virtual world” of interactivity that includes the likes of digital communication, networking, information provision, advertising, entertainment and electronic commerce.
Printed media has been the primary source of communicating information, such as news papers and advertising information, for centuries. The advent and ever-increasing popularity of personal computers and personal electronic devices, such as personal digital assistant (PDA) devices and cellular telephones (e.g., cellular camera phones), over the past few years has expanded the concept of printed media by making it available in an electronically readable and searchable form and by introducing interactive multimedia capabilities, which are unparalleled by traditional printed media.
Unfortunately, a gap exists between the electronic multimedia-based world that is accessible electronically and the physical world of print media. For example, although almost everyone in the developed world has access to printed media and to electronic information on a daily basis, users of printed media and of personal electronic devices do not possess the tools and technology required to form a link between the two (i.e., for facilitating a mixed media document).
Moreover, there are particular advantageous attributes that conventional printed media provides such as tactile feel, no power requirements, and permanency for organization and storage, which are not provided with virtual or digital media. Likewise, there are particular advantageous attributes that conventional digital media provides such as portability (e.g., carried in storage of cell phone or laptop) and ease of transmission (e.g., email).
One particular problem is that a publisher cannot allow access to electronic versions of content using printed versions of the content. For example, for the publisher of a newspaper there is no mechanism that allows its users who receive the printed newspaper on a daily basis to use images of the newspaper to access the same online electronic content as well as augmented content. Moreover, while the publisher typically has the content for the daily newspaper in electronic form prior to printing, there currently does not exist a mechanism to easily migrate that content into an electronic form with augmented content.
A second problem in the prior art is that the image capture devices that are most prevalent and common as part of mobile computing devices (e.g., cell phones) produce low-quality images. In attempting to compare the low-quality images to pristine versions of printed documents, recognition is very difficult if not impossible. Thus there is a need for a method for recognizing low-quality images of printed newspapers.
A third problem in the prior art is that the image recognition process is computationally very expensive and can require seconds if not minutes to accurately recognize the page and location of a pristine document from an input query image. This can especially be a problem with a large data set, for example, millions of pages of documents. Thus, there is a need for mechanisms to improve the speed in which recognition can be performed.
A fourth problem in the prior is that comparing low-quality images to a database of pristine images often produces a number of possible matches. Furthermore, when low-quality images are used as the query image, multiple different recognition algorithms may be required in order to produce any match. Currently the prior art does not have a mechanism to combine the recognition results into a single result that can be presented to the user.
A fifth problem in the prior is that most cell phones and other smart phones have very limited functional capabilities. Most cell phones are used predominately for voice communication, and some add the capability for data communication (simple email), and to capture images. Cell phones use the voice channel, data channel or in some cases WiFi or 3G for data communication, but are very limited as to the functions other than those provided by the manufacturer. While the cell phone industry is growing at a phenomenal rate and more people are accessing online content using their cell phone than ever before, the limited screen size and other capabilities of cell phones are a problem for providing enhanced content and advertising. As a result, advertisers are seeking ways of connecting to these users that do not interfere with normal cell phone use and which minimizes the cost to the cell phone user. There have been attempts to embed advertisements in web pages when browsing the web with a cell phone. These advertisements, however, are based on traditional advertising techniques where only the simple context of the web session along with keywords can be used to generate relevant advertisements. The challenge for search engine companies is to find a way to unobtrusively present users with targeted advertisements while conforming to the small form factor of the cell phone screen.
For these reasons, a need exists for techniques, methods and systems that enable the use of mixed media reality systems in the area of mass media printed publishing.